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We investigate the role of quantum fluctuations in topological quantum phase transitions of quan- 
tum spin Hall insulators and quantum anomalous Hall insulators. Employing the variational cluster 
approximation to obtain the single-particle Green's function of the interacting many-body system, 
we characterize different phases by direct calculation of the recently proposed topological order pa- 
rameter for interacting systems. We pinpoint the influence of quantum fluctuations on the quantum 
spin Hall to Mott insulator transition in several models. Furthermore, we propose a general mech- 
anism by which a topological quantum phase transition can be driven by the divergence of the self 
energy induced by interactions. 
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Introduction. Initiated by the theoretical prediction 
[1] and the experimental discovery [2] of the quantum 
spin Hall (QSH) state featuring a bulk insulating time 
reversal symmetry (TRS) preserving phase with topologi- 
cally protected gapless helical edge modes [HHi] , the field 
of topological insulators (TIs) has been rapidly evolv- 
ing in recent years O |7] . Whereas non- interacting topo- 
logical band insulators are well studied theoretically by 
now, interaction effects substantially complicate the pic- 
ture. One focus of ongoing research hence is to develop a 
deeper understanding of the influence of interactions on 
topological quantum states of matter. 

Along these lines, the topological Mott insulator ^ 
is an example where the non-interacting band struc- 
ture is topologically trivial or metallic, while the interac- 
tions drive the system to the topological insulator phase. 
Recently, realistic materials have been predicted which 
could realize such a phase [9 . In an interacting system, 
one can not use topological invariants based on the single 
particle band states. Instead, a general framework based 
on topological field theory has been proposed, which de- 
fines topological states of matter in terms of the topo- 
logical response function, e.g. the quantized Hall effect 
or the quantized topological magneto-electric effect [lO] . 
These topological response functions can be expressed in 
terms of the single particle Green's function of an inter- 
acting system, and can be defined as the topological order 
parameters (TOP) of the system [TT| - fT5] . The concept of 
the TOP offers a general framework to investigate inter- 
acting topological insulators, and its recently discovered 
simplification [T^ [T3] enables efficient practical calcula- 
tions. Recast in terms of the self energy Y,{k,uj) enter- 
ing the single-particle Green's function, there are both 
static terms that renormalize the bandstructure of the 
non-interacting system as well as contributions from dy- 
namic fiuctuations which affect the pole structure of the 
self energy. Topological quantum phase transitions [T| 



(TQPT) which are due to a critical Dirac point in the 
renormalized band structure can be conveniently encom- 
passed by various mean field approaches. For example, 
the breakdown of the QSH phase in the presence of strong 
Hubbard interactions can be predicted in terms of static 
mean field theory [Tl]. Clear signatures of interaction- 
induced dynamic fluctuations, however, are hard to iden- 
tify unambiguously. In turn, as dynamic fluctuations 
represent genuine interaction effects, they are of crucial 
importance as to further understanding interacting topo- 
logical phases [15] . 

Main results. In this work, we demonstrate the im- 
portance of interaction-driven local dynamic fluctuations 
for TQPTs in two dimensional TIs as well as in quantum 
anomalous Hall (QAH) insulators [121 [IZj, i-G- the lattice 
version of integer quantum Hall (QH) states. Most of 
our findings directly carry over to systems of arbitrary 
spatial dimension. First, we calculate the phase diagram 
of the Kane-Mele-Hnbbard model within the variational 
cluster approximation (VGA) not via probing the exis- 
tence of edge modes in the spectral function, but via di- 
rect calculation of the TOP in the presence of inversion 
symmetry [12j . This approach avoids the necessity of an- 
alyzing edge mode features of the spectral function, and 
might thus be indispensable for topological phases with- 
out edge modes [THl [TH] . Comparing the phase boundary 
obtained from the full VGA calcuation with the static 
limit of our numerical data, we are able to quantitatively 
analyze the infiuence of fiuctuations in the Kane-Mele- 
Hubbard model. By static limit, we mean the large fre- 
quency limit of the self-energy where dynamical fluctu- 
ations accounted for in our full VGA data have decayed 
and only frequency independent Hartree-Fock diagrams 
contribute. Our results explicate that many-body effects 
affect the phase transition from the topologically non- 
trivial to the trivial bands. Second, while in previous lit- 
erature strong interactions were mainly shown to destroy 
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FIG. 1. (Color online) Phase diagram of the Kane-Mele- 
Hubbard model in VGA approximation obtained by direct 
calculation of the TOP in the presence of inversion symmetry 
|12| . The green triangles mark the phase boundary between 
the QSH and a nonmagnetic insulator (NMI) phase if a para- 
magnetic condition is imposed by hand in an 8-site cluster 
VGA calculation. The blue boxes denote the static phase 
boundary between the QSH and an anti-ferromagnetic (AF) 
insulating state in a 6-site cluster VGA calculation. The red 
discs show the phase boundary between the QSH and the 
SDW phase obtained by a self consistent Hatree Fock calcu- 
lation reported in Ref. jl4j. The light pink shaded region 
is our full VGA result for the QSH phase. In the light green 
shaded NMI region, the static result differs from the full VGA 
data. 



the nontrivial topology of a band structure, we show on 
the basis of a toy model how a trivial band insulator can 
be driven into a finite Chcrn number phase by means of 
local dynamic fluctuations. This transition has no effec- 
tive single particle analogue and shows that interactions 
largely expand the scope of topological band structure 
phases. 

Chern numbers and Z2 invariant. The topological in- 
formation of band insulators in the presence of interac- 
tions is encoded in the homotopy of its single particle 
Green's function G in combined frequency-momentum 
space [ini HH [SO]- A perturbative expansion to low- 
est order of the effective action of a gauge field coupled 
to a gapped fermionic system yields the Chern Simons 
term [2TH23] . The prefactor of this term representing 
the Hall conductivity of the system and its higher di- 
mensional analogues is given by a vacuum polarization 
diagram the contribution of which only depends on the 
homotopy class of the fermionic single particle Green's 
function in momentum space [24j [25] . This topological 
coupling constant takes the form 



Cn^Afn duj d^^fcTr {GdG 



, 2n+l 



(1) 



where Mn is a normalization constant ensuring the 
integer-valuedness of C„ and d denotes the exterior 
derivative in combined frequency-momentum space. This 
coupling constant is often times referred to as the topo- 
logical charge of the system and, for a noninteracting sys- 
tem, reduces to the n-th Chern mimber associated with 



its Berry curvature [TO]. Along these lines, the (4-Fl) di- 
mensional time reversal symmetry (TRS) preserving ana- 
logue of the QH effect has been defined [26] . Here, the 
topological charge is given by C2. With the help of di- 
mensional extension, the topological classification of 2D 
and 3D TIs can be reduced to analyzing the (4+1) di- 
mensional quantum Hall phase of the respective extended 
system [TO] [11] . The resulting TOP unifies the notion of 
QH and TI phases from a viewpoint of topology. 

TQPT due to poles of the self energy Whereas TQPTs 
in non-interacting systems are related to Dirac gap clos- 
ing points in the single particle spectrum [T], we en- 
counter a much richer phenomenology in the presence 
of interactions [27l [28] , as they can entail poles in the 
w-dependence of the self energy, i.e. zeros in the Green's 
function. In contrast, the Green's function of an effec- 
tive static band structure is always of the form G{k, to) = 
{iuj ~ H[k))^^ implying that the frequency dependences 
of all effective single particle models are equivalent. Us- 
ing the relation d^ = and the fact that the combined 
frequency-momentum space of the physical system has 
no boundary, the G O G~^ symmetry of Eq. (nj) be- 
comes manifest. Therefore, the poles in the self energy 
which are also poles of G~^ can play a similar role as gap 
closing points, i.e., poles of G in the non-interacting case. 
In general, we are not able to obtain the exact self en- 
ergy of a given problem, but instead impose constraints 
on the functional form along which it becomes accesible. 
A commonly used local self energy approximation reads 



I](fe,cj) = Ei(fc) + S2(u;), 



(2) 



where Si is the previously discussed band structure 
renormalization and E2 is the fiuctuation part which is 
approximated to be independent of momentum fe. De- 
cisive progress on the analysis of fluctuation effects ver- 
sus static mean field effects has been reported recently 

MM- 

TQPT in the Kane-Mele Hubbard model. We now in- 
vestigate the infiuence of local fluctuations as induced by 
on-site Hubbard repulsion on the QSH phase of a honey- 
comb lattice system. The Hamiltonian of the Kane-Mele 
Hubbard model reads 



H = -t 



<*j> 



c\cj + i\ 



<.i,j:S> 
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(3) 



where < • > and <C ■ 3> denote sums over nearest neigh- 
bors and next nearest neighbors respectively, a^ is a 
Pauli matrix in spin space, Ci = {ci^,Cii) is a spinor, 
and I'ij = ±1 depends on whether the orientation of the 
shortest connection between the next nearest neighbors 
is clockwise or counter-clockwise. The variational clus- 
ter approximation (VGA) provides us with the single- 
particle Green's function of the interacting problem. The 
VGA is a quantum cluster approach which describes a 



given model in the thermodynamic hmit via a smaller ex- 
aclty solvable reference frame (cluster) which is then asso- 
ciated with the infinite system within a non-perturbative 
variational scheme [3T1|32]- For two spatial dimensions, 
we are able to keep cluster sizes that allow us to con- 
sider the leading short range correlations on an exact 
footing. For short-ranged interactions, the VGA hence is 
a rather adequate approach for topological band struc- 
tures citeYu2011, WuRachel, racheletal. 

In the absence of disorder and Rashba spin orbit inter- 
action, besides TRS, the model possesses inversion and 
Sz rotation symmetry. Recently, a significant simplifi- 
cation as to the practical calculation of the TOP of a 
TI in the presence of inversion symmetry has been re- 
ported [12], which allows us to efficiently compute the 
phase diagram of our system by direct calculation of the 
TOP: in Fig. [T]we show the phase boundary of the QSH 
phase for various approximations. The static limit of 
our VGA calculation concurs with the antiferromagnetic 
phase boundary. Hence, in this limit, the QSH phase only 
breaks down in favor of antiferromagnetic order implying 
the spontaneous breaking of TRS. Our result is quanti- 
tively different from the results of a static spin density 
wave (SDW) mean field decoupling reported in Ref. [H] 
(red round dots in Fig.fTj). For rather small spin orbit in- 
teraction, i.e. for small A, the phase boundary of our full 
VGA calculation is not determined by the tendency of 
the system towards magnetic order. This indicates that 
at small A where the single particle gap becomes small, 
fluctuations particularly dominate the TQPT from the 
QSH to a nonmagnetic insulating (NMI) phase. This 
qualitatively different behavior as compared to the static 
case is cleary owing to the presence of dynamic fluctua- 
tions, i.e. to the nontrivial oj-dependence of the self en- 
ergy obtained from the VGA calculation. Quantitatively 
similar results for the phase boundary have been previ- 
ously obtained using a complementary approach in VGA 
by probing the appearance of edge modes in the spec- 
tral function |33j . qualitatively matching numerical mi- 
croscopic calculations of the spectral properties via quan- 
tum Monte Garlo methodes which is applicable due to 
particle-hole symmetry in the particular model [Ml ISZ] ■ 

TQPT driven by the self energy. By now, we have only 
discussed an example where interaction effects drive the 
system out of a topologically non-trivial band structure 
into a trivial one. Following Ref. [53] where the self en- 
ergy is assumed scalar in band space, the static mean field 
part and the fluctuation-induced part of the self energy 
decouple and give a product of two integers yielding the 
TOP. Since a nontrivial Z2 invariant, i.e. an odd total 
TOP, can only result as a product of both these factors 
being odd, this rules out the possibihty of a fluctuation 
driven transition from a trivial to a nontrivial state. We 
now investigate whether such a new type of transition 
can still occur when we relax the diagonality constraint 
on the self energy. To this end, let us consider a more 



general matrix valued self energy in Eq. [2] again consisting 
of the Hartree Fock part Si(fc) and the local dynamical 
self energy T,2{uj). The latter can be represented as a 
pole expansion |30| 



E2(w) = V^^(iw-P)-V 



(4) 



with a frequency independent Hermitian N x N matrix 
P, where N is the number of poles of S2('^) on the 
imaginary axis. T^ is an iV x n matrix, where n de- 
notes the number of bands. It has been shown |30j 
that the TOP of the single particle Green's function 
G{k,uj) = {iu! + fi — h{k) — T,{k,uj))~'^ with the single 
particle Bloch Hamiltonian h{k) can be calculated by in- 
troducing an effective extended {n + N) x (n + N) single 
particle Hamiltonian 



H{k) 



h{k) 



Si(fc) 

V 



p 



(5) 



and then calculating the TOP of the single particle 
Green's function G{k,uj) — {iu — H{k))~^. Note that a 
factorization into a non-interacting and fluctuation part 
is not manifest in this more general case. We now show 
that a transition from trivial to nontrivial can in princi- 
ple be driven by means of local fluctuations as described 
by a nonscalar local self energy. While our idea is very 
general, let us for concreteness explicitly construct a min- 
imal toy model for such a type of transition. Gonsider 
the two band model of the QAH insulator [T7] 



h{k) = v'{k)ai, 



(6) 



where v^ = sin(fca;), f^ = sin(/cj,), v"^ = (to + CQ^ik^) + 
cos(fcy)) and ?Ti is a real parameter which tunes the sign 
of the band gap. Assuming an interaction which brings 
about two poles in the self energy, we make the following 
ansatz for the parameters V, P of the pole expansion 

y = /ia„ P = \gz (7) 

which, for the effective extended Hamiltonian, yields 



Bik^ 



h{k) 



Act, 



(8) 



For TO, — —2.5, the two band model is located in its trivial 
regime. A two dimensional phase diagram with single 
particle Hamiltonian defined in Eq. ((sl) is shown in Fig. [2] 
We find that local fluctuations can drive a system into a 
topologically nontrivial phase in this model. Note that 
Si(fc) = in our case. Hence, exclusively I]2(w) causes 
a TQPT which has no band structure analogue due to 
a gap closing in the two band Hamiltonian Q, and is a 
pure fluctuation-induced phase transition that does not 
connect to a static mean field ansatz. 

Discussion. The two fluctuation-induced transitions 
we describe allow us to gain a deeper understanding of 
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FIG. 2. (Color online) Phase diagram of the QAH insulator 
with local fluctuations, m — —2.5 is fixed. The pole structure 
of the local self energy encoded in A, fi is varied. The Chern 
numbers Ci of the different phase domains are indicated. The 
black regions denote numerically unstable transition regimes. 



the role of interactions in topological band structures. 
As shown above, in a fluctuation-driven TQPT, G and 
G^^ switch roles as compared to a transition with a crit- 
ical Dirac-point in the static single particle spectrum. 
Here, it is the pole of the self energy and not the quasi- 
particle pole which determines the TQPT. Our model 
with non-trivial Chern number induced by interaction is 
an explicit example for this mechanism. 

To summarize, we have identified the fingerprints of 
local interaction-induced dynamical fluctuations in two 
different kinds of TQPTs. First, we have investigated 
fluctuation-driven contributions to the critical Hubbard 
interaction strength of the QSH-Mott transition in the 
Kane-Mele-Hubbard model at the level of the TOP. In 
our VGA calculation, the phase boundary obtained from 
the static limit of the single particle Green's function dif- 
fers qualitatively from the full VGA data: whereas in the 
static limit the QSH phase breaks down only in favor of 
magnetic order, dynamic fluctuations retained in the full 
VGA result lead to a transition between the QSH and a 
NMI phase for smaller interactions. Second, a proof of 
principle has been given for a reverse transition from a 
trivial band insulator to a topologically non-trivial QAH 
insulator, where the pole structure of the frequency de- 
pendent self energy causes a critical singularity which 
is substantially different from a critical Dirac point in 
a band inversion TQPT caused by spin orbit coupling. 
The fluctuation-induced effects have no band structure 
analogue and are a genuine signature of electronic cor- 



relations beyond mean field theory. Our arguments also 
carry over to three spatial dimensions, where it will be 
interesting to consider our findings in the framework of 
interaction effects for 3-D TI's and Weyl semimetals |38) . 

Note added. When this manuscript was finished, we 
became aware of complementary independent results re- 
ported in Wang et al, EPL 98, 57001 (2012). 
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